Cellscope apparatus and methods for imaging

ABSTRACT

An improved system and methods for enhancing the imaging of cameras included with wireless mobile devices, such as cellular phone or tablets. The imaging system includes a releasable optical attachment for imaging skin surfaces and cavities of the body. The releasable optical attachment comprises optical enhancement elements such as magnifying lenses, illumination diverting elements, and filters. Images can be viewed and analyzed on the mobile device, or transmitted to another location/device for analysis by a person or software. The results can be used to provide diagnosis, or for a variety of other applications including image comparison over time and product recommendations.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/052,347 filed on Feb. 24, 2016, incorporated herein by reference inits entirety, which is a continuation of U.S. patent application Ser.No. 13/855,501 filed on Apr. 2, 2013, now U.S. Pat. No. 9,325,884,incorporated herein by reference in its entirety, which is a 35 U.S.C.§111(a) continuation of PCT international application numberPCT/US2011/058466 filed on Oct. 28, 2011, incorporated herein byreference in its entirety, which claims priority to and the benefit ofU.S. provisional patent application Ser. No. 61/532,617 filed on Sep. 9,2011, incorporated herein by reference in its entirety, and which claimspriority to and the benefit of U.S. provisional patent application Ser.No. 61/408,568 filed on Oct. 29, 2010, incorporated herein by referencein its entirety. This application is also a division of U.S. patentapplication Ser. No. 13/855,501 filed on Apr. 2, 2013, now U.S. Pat. No.9,325,884, incorporated herein by reference in its entirety. Priority isclaimed to each of the foregoing applications.

The above-referenced PCT international application was published as PCTInternational Publication No. WO 2012/058641 on May 3, 2012 andrepublished on Jul. 5, 2012, and is incorporated herein by reference inits entirety.

This application is related to U.S. patent application Ser. No.12/826,375 filed on Jun. 29, 2010, now U.S. Pat. No. 8,743,194,incorporated herein by reference in its entirety, which claims priorityfrom, and is a 35 U.S.C. §111(a) continuation of, PCT internationalapplication number PCT/US2008/088646 filed on Dec. 31, 2008,incorporated herein by reference in its entirety, which claims priorityfrom U.S. provisional application Ser. No. 61/018,537 filed on Jan. 2,2008, incorporated herein by reference in its entirety.

This application is also related to PCT International Publication No. WO2009/088930 published on Jul. 16, 2009, incorporated herein by referencein its entirety.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

Not Applicable

INCORPORATION-BY-REFERENCE OF COMPUTER PROGRAM APPENDIX

Not Applicable

NOTICE OF MATERIAL SUBJECT TO COPYRIGHT PROTECTION

A portion of the material in this patent document is subject tocopyright protection under the copyright laws of the United States andof other countries. The owner of the copyright rights has no objectionto the facsimile reproduction by anyone of the patent document or thepatent disclosure, as it appears in the United States Patent andTrademark Office publicly available file or records, but otherwisereserves all copyright rights whatsoever. The copyright owner does nothereby waive any of its rights to have this patent document maintainedin secrecy, including without limitation its rights pursuant to 37C.F.R. §1.14.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention pertains generally to imaging devices, and moreparticularly to devices and methods for enhanced imaging with mobileelectronics devices.

2. Description of Related Art

Handheld mobile devices, such as cellular phones, tablets, PDA's, etc.,are becoming increasingly useful for imaging due to their readyavailability to communicate with other devices wirelessly. However, thecameras and illumination sources included with most mobile electronicdevices are primitive with respect to the type of imaging that may berequired for special surfaces such as a patient's skin or body passage.

Many common medical tests may be performed using telemedicine, butstand-alone devices used by physicians are often too expensive orspecialized to appeal to consumers.

Accordingly, an object of the present invention is an apparatus thatallows mobile devices to perform as enhanced camera's and telemedicinetools, utilizing their familiar interface and ease of image capture andtransmission. At least some of these objectives will be met in thedescription below.

BRIEF SUMMARY OF THE INVENTION

The present invention is an improved system and method for enhancedimaging using wireless transmission devices with a camera (such as amobile phone) combined with an optical attachment. Images can be viewedand analyzed on the mobile device, or transmitted to anotherlocation/device for analysis by a person or software. The results may beused to provide diagnosis, or for a variety of other applicationsincluding image comparison over time and product recommendations.

One aspect of the invention is an imaging apparatus for a portablewireless device having a built-in camera. The apparatus includes areleasable optical assembly having a housing comprising an attachmentsurface for releasably coupling the releasable optical assembly to theportable wireless device, and an optical transmission element. Theoptical transmission element is configured to enhance an image taken bythe built-in camera prior to the image being received the portablewireless device.

Another aspect is a system for enhancing and post-processing imagesobtained from a portable wireless device having a built-in camera,comprising: a releasable optical assembly, comprising: a housingcomprising an attachment surface for releasably coupling the releasableoptical assembly to the portable wireless device, and an opticaltransmission element. The optical transmission element is configured toenhance an image taken by the built-in camera prior to the image beingreceived the portable wireless device. The system further comprisesprogramming executable on said wireless device or other external devicefor receiving the enhanced image and post processing the enhanced image.

Further aspects of the invention will be brought out in the followingportions of the specification, wherein the detailed description is forthe purpose of fully disclosing preferred embodiments of the inventionwithout placing limitations thereon.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING(S)

The invention will be more fully understood by reference to thefollowing drawings which are for illustrative purposes only:

FIG. 1A is a view of the imaging apparatus of the present inventionpositioned adjacent mobile device.

FIG. 1B shows the imaging apparatus of FIG. 1A installed on a mobiledevice in accordance with the present invention.

FIG. 2 shows a view of the rear side of the imaging apparatus of FIG.1A.

FIG. 3 shows the imaging apparatus of FIG. 1A installed on a mobiledevice in accordance with the present invention, with outer speculumremoved for clarity.

FIG. 4 shows a cross-sectional view of the imaging apparatus of FIG. 1Ainstalled on a mobile device in accordance with the present invention.

FIG. 5 shows an alternative imaging apparatus in accordance with thepresent invention.

FIG. 6 shows the imaging apparatus of FIG. 5 with the diffuser removedfor clarity.

FIG. 7 shows a rear view of the housing of the imaging apparatus of FIG.4.

FIG. 8 shows a cross-sectional view of the imaging apparatus of FIG. 5installed on a mobile device in accordance with the present invention.

FIG. 9 shows the imaging apparatus of FIG. 5 with calibration module inaccordance with the present invention.

FIGS. 10A and 10B show perspective and plan views, respectively, of anexposure calibration tool according to the present invention.

FIG. 11 shows a perspective view of a focus calibration tool accordingto the present invention.

FIG. 12A illustrates a side view of an embodiment for coupling theoptical attachment of the present invention.

FIG. 12B illustrates a plan view of an alternative embodiment forcoupling for the optical attachment of the present invention.

FIG. 13 shows a side view of an embodiment of a modular imaging systemcomprising components for imaging and illumination according to theinvention.

FIG. 14 shows embodiments of a modular attachment system according tothe present invention.

FIG. 15 shows a method for two-way optical data transmission from amobile device according to the present invention.

FIG. 16 shows a flow diagram of the image normalization step of FIG. 15.

FIG. 17 shows a flow diagram of the image analysis step of FIG. 15.

DETAILED DESCRIPTION OF THE INVENTION

Referring more particularly to the figures, the present invention can beembodied in various ways.

FIG. 1A illustrates a perspective view of an imaging apparatus 10positioned adjacent handheld mobile electronic device 12. Mobile device12 is illustrated as an iPhone 4 in the various figures depicted herein.However, it is appreciated mobile device 12 may be any wireless-enableddevice having a camera, i.e. mobile device 12 may comprise any number ofpossible makes of cellular phones, PDA's, tablets, etc.

FIG. 1B shows the imaging apparatus 10 installed on the mobile device 12in accordance with the present invention. Imaging apparatus 10 includesone or more optical transmission elements (e.g. lens, fiber optics,etc.) that are configured to enhance an image received by a camera 22 ofdevice 12 prior to that image being received by the camera 22 CCD (notshown). The optical transmission elements may provide magnificationand/or improved illumination to the imaged target, in addition to otherimaging enhancements.

Imaging apparatus 10 includes a base member 14 that can be coupled tothe mobile device 12. FIGS. 1A and 1B illustrate “clip-on” couplingmeans or surface 20, although other removable means such as slide-on,snap-on, or adhesives/adhesive backings are also contemplated. Whilebase member 14 is shown in the current figures as a partial cover of themobile device 12, base member 14 may also comprise a full case or cover(i.e. extending along the length of phone 12) that covers one or moresurfaces, and releasably couples to the mobile device 12.

FIGS. 1A and 1B show a releasable optical assembly 16 attached to base14. The releasable optical assembly 16 comprises a cylindrical housing40 and a conical-shaped speculum 18. Speculum 18 is shown sized andshaped for ear drum imaging as an otoscope. The speculum 18 is one ofmany different optical attachments that may be used to interface theimaging apparatus 10 with varying anatomical features. In particular,speculum 18 may be sized and shaped for proper interface with variousbody cavities, including nasal, oral, vaginal, and anal cavities, etc.Speculum 18 is configured for quick release attachment to housing 40. Asshown in FIG. 4, the speculum 18 is shown with a contact fit overhousing 40. However, other attachment means, e.g. mating threads, clips,snaps, tab-and-grooves, etc., may also be used.

Accordingly, the base member 14 also includes means (not shown) forcoupling the releasable optical assembly 16 to the base member 14, suchas clip-on, snap-on, slide-on, twist-on, and other conventional meansfor attachment. For example the releasable optical assembly 16 may havepins that line up with and snap into one or more corresponding slots inthe base member, or vise versa.

FIG. 2 shows a view of the rear side of the imaging apparatus 10 andbase 14. Base 14 comprises clip-on walls 20 that help retain the base 14on to the mobile device 12. The base 14 comprises an illumination port34 configured to align with the phone's LED flash 24 to allowilluminated light to pass through the case and into the releasableoptical assembly 16. The base 14 also comprises and an imaging port 32allowing line of sight for the phone's camera 22 from the releasableoptical assembly 16.

FIG. 3 shows the imaging apparatus 10 installed on a mobile device 12with outer speculum attachment 18 removed for clarity. An optical tube56 is centered within the housing opening and has a tube aperture 58configured to line up with the imaging port 32 of the base 14.

FIG. 4 shows a cross-sectional view of the imaging apparatus 10installed on mobile device 12. The back wall 68 of the housing 40comprises an imaging opening 42 that is configured to line up withoptical tube 56, imaging port 32 and camera 22. Optical tube 56 may bean integral member with housing 40, or attached to the housing 40. Theimaging port 32 and opening 42 provide an optical path for imaging. Alens 62 (or series of lenses) is preferably disposed within the opticaltube 56, imaging opening 42 or imaging port 32 (shown in preferredconfiguration in imaging opening 42). The lens 62 is preferablyconfigured to provide magnification of the target anatomy (e.g. eardrum, etc.). In one embodiment, lens 62 comprises a plano-convex lenswith a 48 mm focal length. It is appreciated that lens 62 may compriseany number of differing types, (e.g. bi-convex, convex-meniscus, etc.),or any combination of lenses known in the art.

It is also appreciated that lens 62 may comprise micro-lens array (e.g.two or more lenses space apart in a planar array) in place of a singlelens. Light passes through normal to the 2D array (not shown). Each lenselement in the array has a shorter focal length/higher magnificationthan the larger lenses (e.g., the 48 mm f lens in the otoscope), so thedesign can be much more compact. Due to the space in between the arrayelements, the array can be scanned to capture the individual images,which are then assembled into a unified image of the field without gaps.This allows for a high magnification system in a much smaller formfactor.

The imaging apparatus 10 preferably comprises means for directing thelight from the phone's LED flash 24 to provide improved illumination.Use of just the LED flash 24 to illuminate the anatomical target (e.g.ear drum) directly would lead to shadows from the optical tube 56.Illumination with the fiber optics may be used to eliminate shadows. Thelight can be directed to provide on-axis illumination, as well asoblique or off-axis illumination for better contrast for someapplications.

The back wall 68 of the housing 40 further comprises an illuminationopening 44 that is configured to line up with a fiber optic bundle 50 a,illumination port 34 and flash 24. Fiber optic bundle 50 a generallycomprises one or more optical fibers, configured with end glow or sideglow characteristics to suit the particular application or anatomy beingimaged.

In a preferred embodiment, the fiber optic bundle 50 a connects at oneend to the illumination opening 44 and wraps and extends to optical tube56, with optical fibers 50 extending axially around optical tube 56 toform a coaxial layer at least partially surrounding the optical tube.The individual optical fibers 50 preferably form a continuous layeraround tube 56. However it is appreciated that the optical fibers mayalso surround the tube at spaced intervals. Optical fibers 50 extend atleast a portion of the length of tube 56 such that the end of theoptical fibers are aligned in the direction of the tube 56 axis andpreferably near the free end of the tube 56. Optical fibers are held inplace on tube 56 via an adhesive, band 47, or like attachment means. Thebundle 50 a and fibers 50 are configured to propagate light from flash24 (e.g. LED or the like) and direct it toward the free end of theoptical tube 56 and in the direction of speculum 18. The optical fibers50 arrayed around the end of the tube 56 provide a more uniform ring ofillumination in the direction of the target anatomy. Such directed lightis particularly useful for viewing/imaging cavities in the body such asthe ear or mouth, where light is limited.

The imaging apparatus is configured such that when base member 14 andreleasable optical assembly 16 are attached to device 12, all opticalapertures (speculum aperture 19, optical tube aperture 58, opticalopening 42, and optical port 32) are in substantially concentricalignment with camera 22, and correspondingly, all illuminationapertures (illumination opening 44, illumination port 34) are all insubstantially concentric alignment with flash 24.

In a preferred embodiment, the optical assembly 16 may also contain oneor more filters in the illumination path to improve the illumination ofthe target anatomy. The filters may be positioned in fixed or adjustableconfiguration at or between one or more structures of the imagingapparatus 10. A sliding color spectrum filter may also be positioned inthe path of the LED flash 24 to enable selective color illumination forspectrophotometric applications.

In one embodiment, the filters may comprise one or more of polarizingfilters, neutral density filters, or diffusers, or the like. Forexample, one or more polarizing filters can be used to reduce glareand/or shadows, in addition to controlling illumination intensity anduniformity. Thus, filters may be used to control the intensity andpattern of illumination based on the phone's LED flash 24 (or anintegrated light source as provided in FIG. 12, described in furtherdetail below), and/or to reduce glare for imaging a reflective sample(such as like oily skin). With use of variable filters, the intensity ofsample illumination can be adjusted without changing the output providedby the device LED 24. For the iPhone for example, the LED flash 24 iseither on or off when no software support for adjustable intensity isprovided, making exposure difficult in some applications.

In one example of a tunable illumination system illustrated in FIG. 4,two polarizing filters 52, 54 are stacked in series between the handsetLED 24 and the illuminated anatomical target. Referring further to FIG.7, the first filter 52 is fixed and disposed within a rectangularcounter bore 46 set into the rear wall 68 of housing 40. The secondfilter 54 is set into a circular counter bore 48 in housing 40, and isconfigured to be rotated with use of lever 66 to modulate the intensityof the light transmitted to the sample anatomy. It is appreciated thatthe filters 52, 54 may be positioned anywhere within the optical path,e.g. within corresponding counter bores (not shown) in the base 14. Theintensity of the transmitted light is controlled by the relativeorientation of the filters 52, 54, where parallel orientation permitsthe maximum transmission and perpendicular orientation permits minimaltransmission. This could also be achieved with a gradient neutraldensity filter, which could be moved (rotated in the case of a circulargradient) to adjust the light transmission

A spectral selection tool may also be included for controllingillumination wavelength. A slider (or color wheel, etc., not shown)containing color filters may provide for specific wavelengths to bechosen from the LED 24 illumination. This enables comparison of imagestaken with different colors to look for absorption differences. This isuseful in skin imaging, for example.

The optical assembly 16 may also comprise one or more filters in theoptical path (e.g. between optical port 32 and optical opening 42). Forexample, one or more filters comprising a polarizer, neutral densityfilter, or spectral selection filter may be positioned in the opticalpath using a system similar to the illumination filters 52, 54 of FIG.4.

Optical assembly 16 may further allow for adjustment of the opticalcomponents (e.g. lens 62). For example, optical assembly 16 may comprisea lens slider (not shown) similar to slider 66 in FIGS. 4-6 to allow forone or more lenses to be moved in and out of the optical path (lensstacking), or for lens position to be moved along the optical axis.Removing all attachment lenses 62 from the path provides the device's 12standard imaging characteristics, while adding one or more lenses 62 inthe optical path modifies the imaging characteristics to suit theapplication. Thus, single optical assembly 16 may be configured toprovide variable magnification and numerical aperture characteristics,similar to the way that many microscopes contain a turret of objectives.

FIGS. 5-8 show an imaging apparatus 25 preferably configured for use inimaging skin surfaces, and in particular, non-cavity-type skin surfaces.Referring to FIG. 5, imaging apparatus 25 includes a releasable opticalassembly 17 attached to base 14. The releasable optical assembly 17comprises an open ended housing or spacer 45 having an optical opening42. As shown in further detail in the cross-sectional view of FIG. 8,the optical opening 42 provides an open path to the optical port 32 ofbase 14 and camera 22 of device 12. Spacer 45 of imaging apparatus 25provides a set distance between camera 22, lens 62 and the image target.When the imaging apparatus 25 is touched to the image target (skin,hair, etc.) directly, the image target is within the focus range for thecamera 22, facilitating image collection. This distance may beadjustable (to discrete distances or continuously) to accommodatedifferent imaging applications. Spacer 45 may also include an additionalsliding tube (as provided in tube 96 in calibration tool 90 shown inFIG. 11) to provide adjustable spacing. Thus, the spacer element 45could be adjusted to discrete or continuous height steps to accommodatethe focal length of the lens or lenses 62.

Fiber optic bundle 50 b may be used to change the shape and/or positionof the light source 24. As shown in FIGS. 5 and 6, fiber optic bundle 50b could be arrayed under a diffuser 60 to provide controlled (e.g.uniform) illumination, (FIG. 6 shows the diffuser removed for clarity).Referring now to the cross-sectional view of FIG. 8, the fiber opticbundle 50 b is disposed within illumination opening 44 of the housing45, such that the fiber optic bundle 50 b is in line with illuminationport 34 of the base 14 and LED 24 of device 12. Similar to apparatus 16shown in FIGS. 1-4, and 7, the optical attachment assembly 17 may alsocomprise filters 52, 54, and slider handle 66 in addition to bores 46and 48 in the housing 45 to provide additional illumination filtering.

FIG. 9 shows the imaging apparatus 27 similar to imaging apparatus 25 ofFIGS. 5-6, with calibration module 70 disposed on spaced 45. Calibrationmodule 70 comprises a test pattern feature 72 which appears in a portionof the image field of view to enable calibration of color, whitebalance, exposure, and size. In one embodiment, black, white, grey andcolor patches with known color values on the calibration pattern serveas a reference for post-processing of the image. The test pattern alsoprovides an exposure reference (based on the appearance of the blackpatches in the image) and a size scale due to the known size of thepattern elements. The calibration module 70 is instrumental incontrolling imaging from a device 12, such as a mobile phone, which wasnot designed to produce absolute color/exposure references. Preferably,the calibration module 70 is integrated into the imaging apparatus 27.It is also appreciated that calibration module 70 may be disposed withinaperture 19 of disposable speculum 18 for calibrating device 10.

Alternatively, a reference feature may be used as an external testpattern or a common object (for example, a coin). Including a knownobject in the field of view enables size and color calibration of theresulting image, especially when its depth position is known or can bedetermined from the image. A coin held next to the earlobe can be usedby software to determine (or reasonably estimate) the absolute size ofthe facial features.

FIGS. 10A through 11 illustrate focus and illumination tools forpre-capture calibration to pre-set the exposure and focus beforecapturing an image.

FIGS. 10A and 10B show perspective and plan views, respectively, of anexposure calibration tool 80 according to the present invention. In theotoscope application, for example, the camera's 22 exposure setting mayuse a larger region of interest (ROI) than the actual image region atthe end of the tip of speculum 18. In that case it is useful to set theexposure relative to a brighter image in the smaller field at the end ofthe tip, to compensate for the mismatch with the camera's 12 ROI.Exposure calibration tool 80 comprises a stepped platform 82 a distanceh from a lower platform 84. The stepped platform 82 has large 86 andsmall 88 dark apertures, and the lower platform 84 correspondinglycomprises large 76 and small 78 lighter apertures. Exposure calibrationis performed by inserting imaging apparatus 10, 25 (e.g. otoscope tip18) at the particular apertures 76, 78, 86, 88. The stepped platform 82provides a sharp image at a known distance to pre-set focus, the lowerplatform 84 is bright (e.g. with matte aluminum foil or like material,or adjustable to material with a range of known reflectance values) topre-set the exposure.

It is also useful to preset the focus so that the camera 22 is focusedon a known distance before imaging a delicate area of anatomy, such as apatient's eardrum. FIG. 11 shows a perspective view of a focuscalibration tool 90 having a base 92 and sliding tube 96 disposed withinbase tube 94. The sliding tube adjusts to various heights H by slidingwithin base tube height to allow for a range of focal distance pre-sets.The calibration tool comprises an illumination opening 95 and imagingaperture 98.

The tools 80 and 90 to calibrate focus and exposure may be a separateparts (as shown in FIGS. 10A through 11) or integrated within theimaging device (e.g. calibration module 70 in device 27 FIG. 9). Thecalibration tool may be adjustable to allow for a range of focaldistances (as shown), or use a color wheel or other means to adjust theexposure calibration area

FIGS. 12A through 13 illustrate different attachment means according tothe present invention.

FIG. 12 A illustrates a side view attachment means coupling an imagingapparatus 104 via manually positioning and affixing the imagingapparatus 104 with a ring 108 (which may be replaceable). Ring 108 maycomprise adhesive tape, hook-and-loop closure or magnet to releasablyattach to the phone.

FIG. 12B illustrates a plan view of an alternative embodiment forcoupling assembly 100. Coupling assembly 100 comprises spring-loadedpincher attachment 102 that is configured to be silideably positioned inthe vertical direction. Coupling assembly 100 further comprises arectangular slot 106 that allows the imaging assembly 104 to be adjustedin the horizontal direction to coincide with the phone 12 camera 22 andflash 24.

FIG. 13 illustrates modular imaging system 120 comprising separatedcomponents for imaging and illumination. The flexible fiber bundle 124is attached to the optical cone 122, such that the fiber bundle 124 canbe bent to accommodate a variety of camera 22 and LED/flash 24 positionson the mobile device 12 or case 14 provided with the device. The system120 is shown in the otoscope-type configuration 10 of FIGS. 1-4, with acone 122 to which disposable specula can be attached. The same modularapproach may also be applied to the apparatus 25 of FIGS. 5-6.

FIG. 14 shows a modular attachment system 150 having independent lightsource 162. System 150 may comprise a disposable speculum 152 thattapers toward its tapered free end, and having tabs 164 at its proximalend for attachment to slot 166 in housing 168. A separate attachmentcone 154 may also be included, and have slot 166 for attaching tohousing 168. An optical tube 156 containing one or more lenses 62 isprovided within housing 168, and is wrapped in fiber optics 158 forillumination. The fiber bundle 158 can draw light from the phone's LEDflash 24, an external light source 162 by means of elongate tube 160.

The image enhancement systems shown in FIGS. 1-14 may be used not onlyfor capturing enhanced digital images and video, but also as a real-timescope for viewing patient anatomy. For example, one could be treatingthe integrated mobile device 12 as an otoscope for display on the mobiledevice's screen. At any time, an image capture or video may also beinitiated to store the image or video in memory, and transmit the imageor video if necessary.

FIGS. 15 through 17 illustrate a system and method for post processingan image captured from a mobile device 12. While it is preferred thatsoftware for carrying out method 200 be used in conjunction with theimage enhancement apparatus disclosed in FIGS. 1 through 14, it isappreciated that the methods disclosed in FIGS. 15 through 17 may beused for any mobile device, webcam, or camera directly, with or withoutthe image enhancement apparatus disclosed in FIGS. 1 through 14.Accordingly, a preferred embodiment of the present invention is a systemcomprising one or more of the enhancement apparatus disclosed in FIGS. 1through 14, along with software using methods and algorithms embodied inFIGS. 15 through 17.

FIG. 15 shows a method 200 for two-way optical data transmission from amobile device, which allows the user to point the imaging apparatus 10,25 at a surface, while displaying the real-time image of the surface ona secondary device, such as a computer. The secondary device can controlthe features of the imaging system 10, 25, such as choosing the focusregion of interest, exposure, and image collection, etc. Method 200 maybe implemented within a software module or application for execution ona processor of the wireless device, as well as applications forexecution on the secondary device such as a computer.

As shown in FIG. 15, a first step in the method 200 is calibration ofthe camera 22 at block 202. Next an image is captured at block 204 andtransmitted to secondary device at block 206. Image normalization isthen performed at block 208, and image analysis and/or diagnosis isperformed at step 210. Captured images may be transmitted for analysisby software algorithms or a person for a range of applications includingdiagnosis, monitoring and product recommendations.

While image transmission step may be desirable to perform calculationintensive tasks such as image normalization step 208 and imageanalysis/diagnosis step 210 on a computer or other device configured toquickly perform such calculations, is appreciated that transmission step206 is optional, and that one or both of the image normalization step208 and image analysis/diagnosis step 210 may be performed usingsoftware and programming on the mobile device 12.

FIG. 16 illustrates in further detail the image normalization step 208of method 200. First, exposure and focus sharpness of image 212 iscalculated with a known feature at stem 214. If the calculation returnsnon-acceptable results at evaluation step 218, the device 12 is promptedfor another image to be captured at step 216. If the calculation returnsnon-acceptable results at evaluation step 218, the reference data fromimage is used at step 220. The image is then adjusted at step 220 togenerate the normalized image 222.

Image normalization step 208 has the capability to normalize the imagebased on test pattern portion of the image 212 as an application loadedas either on-board software on the mobile device 12, or by uploading theimage 212 for server-side processing. The normalization step 208, whichmay comprise white balance, exposure, focus, etc., may be based on theimaging apparatus 10 integrated test pattern (e.g. calibration module70), or on another known standard. The orientation of the image 212 mayalso be adjusted in step 210 using reference features in the image 212(e.g. in the case of ear drum imaging, adjusting the rotation of theimage based on the “cone of light” feature of the ear drum), or usingon-board sensors within device 12 such as the accelerometer orgyroscope. This is a significant advantage in ear drum imaging, forexample, where the doctor is used to a certain orientation but theimaging device may produce a rotated view. Normalization step 208 canidentify corresponding features from two images to determine whetherthey have overlapping areas. This too can be improved by adding inaccelerometer/gyroscope data, so that the overlap detection is based onimages with the same orientation (this compensates for a rotationdifference between the two images).

Normalization data may also be used to override the camera 22 automaticfocus and exposure settings so as not to interfere with the acquiredcalibrated settings.

FIG. 17 illustrates in further detail the image analysis and/ordiagnosis step 210 of method 200. Diagnosis step 210 provides imageprocessing (either via executing an application on-board thephone/device 12 or uploaded for server-side processing) to analyzeimages for features of interest, such as skin image analysis forwrinkles, lines, redness, blood vessels, dryness, coloration, UV damageand roughness. At step 234, the normalized image 222, similar imagecharacteristics (e.g. skin type) identified from other users 230 andother image data and results from image analysis or other users areinput. Data from 230 and 232 may comprise a database of images andassociated data. At step 236 the normalized data 222 is compared withdata from past images or other user data to arrive at resulting image238.

Software loaded on the mobile device 12 or secondary device may use theresults 238 of image analysis to provide recommendations and/or targetedads to user. In the case of skin care, image analysis provides data onskin type and condition (e.g. Fitzpatrick type, oily/dry/combination, UVdamage, wrinkles, etc) and the user is provided with targeted productrecommendations. The software also allows users to track images overtime to monitor changes, results of product usage, etc. Recommendationsand ads can be based on aggregated data from users from a similar class(e.g. with similar skin types). For example, image analysis (from themagnified imaging system 10, 25 of the present invention, or standardcameras) may be used to show that users with similar skin have recordedbetter results using a particular product.

For skin imaging of the face, which can be difficult to self-take, themethod 200 and associated software have the capability to transmit theimage in real time for display on the secondary device (e.g., anotherphone or computer via the device application or website). Thecommunication can be two-way, so that the imaging system functions(focus, exposure, image capture, etc.) can be controlled from the seconddevice.

Images 204 may be tagged with GPS and/or time/date stamp information,which can be used to authenticate the user or perform geographicalmeta-analysis of results.

Video may also be captured by on the device 12, and analyzed by method200 using software on the device or server-side to identify key framesor to generate an enhanced single frame. For example, in the ear examapplication, the software may identify the best frame to present to thephysician by using a standard sharpness detection algorithm to find theframe with the sharpest cone of light (bright reflection off the eardrum with a defined border). This video could be taken while moving thedevice around, or while adjusting the focal distance (such as runningthe camera's 12 autofocus function). Video during autofocus provides aset of frames that are focused on a series of focal distance planes.This may then be used to identify the best frame, or to providesomething analogous to the 3D image reconstruction done with a confocalmicroscope image stack.

It is also appreciated that the optical enhancement systems of FIGS.1-14 and image processing methods of FIGS. 15-17 may also be useful with3D stereo cameras now available on some phones). Stereo images may beused help in skin imaging, for example to measure the depth of wrinklesor the height of a mole.

Embodiments of the present invention may be described with reference toflowchart illustrations of methods and systems according to embodimentsof the invention, and/or algorithms, formulae, or other computationaldepictions, which may also be implemented as computer program products.In this regard, each block or step of a flowchart, and combinations ofblocks (and/or steps) in a flowchart, algorithm, formula, orcomputational depiction can be implemented by various means, such ashardware, firmware, and/or software including one or more computerprogram instructions embodied in computer-readable program code logic.As will be appreciated, any such computer program instructions may beloaded onto a computer, including without limitation a general purposecomputer or special purpose computer, or other programmable processingapparatus to produce a machine, such that the computer programinstructions which execute on the computer or other programmableprocessing apparatus create means for implementing the functionsspecified in the block(s) of the flowchart(s).

Accordingly, blocks of the flowcharts, algorithms, formulae, orcomputational depictions support combinations of means for performingthe specified functions, combinations of steps for performing thespecified functions, and computer program instructions, such as embodiedin computer-readable program code logic means, for performing thespecified functions. It will also be understood that each block of theflowchart illustrations, algorithms, formulae, or computationaldepictions and combinations thereof described herein, can be implementedby special purpose hardware-based computer systems which perform thespecified functions or steps, or combinations of special purposehardware and computer-readable program code logic means.

Furthermore, these computer program instructions, such as embodied incomputer-readable program code logic, may also be stored in acomputer-readable memory that can direct a computer or otherprogrammable processing apparatus to function in a particular manner,such that the instructions stored in the computer-readable memoryproduce an article of manufacture including instruction means whichimplement the function specified in the block(s) of the flowchart(s).The computer program instructions may also be loaded onto a computer orother programmable processing apparatus to cause a series of operationalsteps to be performed on the computer or other programmable processingapparatus to produce a computer-implemented process such that theinstructions which execute on the computer or other programmableprocessing apparatus provide steps for implementing the functionsspecified in the block(s) of the flowchart(s), algorithm(s), formula(e),or computational depiction(s).

From the discussion above it will be appreciated that the invention canbe embodied in various ways, including the following:

1. An imaging apparatus for a portable wireless device having a built-incamera, comprising: a releasable optical assembly, comprising: ahousing; the housing comprising an attachment surface for releasablycoupling the releasable optical assembly to the portable wirelessdevice; and an optical transmission element; wherein the opticaltransmission element is configured to enhance an image taken by thebuilt-in camera prior to the image being received the portable wirelessdevice.

2. The apparatus of embodiment 1, wherein the optical transmissionelement comprises a first optical transmission element configured tomagnify the image.

3. The apparatus of embodiment 2, wherein the optical transmissionelement comprises a second optical transmission element configured toenhance illumination of the image.

4. The apparatus of embodiment 3: wherein the first optical transmissionelement comprises a lens; the housing configured to house the lens in analignment within an optical path of the built-in camera.

5. The apparatus of embodiment 4: the portable wireless device furthercomprising an illumination source; wherein the second opticaltransmission element comprises an optical fiber; the housing configuredto house the optical fiber in an alignment within the illumination pathof the illumination source; and wherein the optical fiber is configuredto alter the illumination path to enhance illumination of the image.

6. The apparatus of embodiment 5: the releasable optical assemblyfurther comprising one or more optical filters coupled to the housing;and the housing configured to house the one or more optical filters inan alignment within the one or more of the illumination path or opticalpath.

7. The apparatus of embodiment 6, wherein the one or more opticalfilters comprises a filter comprising one or more of a: polarizer,neutral density filter, or spectral selection filter.

8. The apparatus of embodiment 6, wherein the one or more opticalfilters comprises: a first filter having a fixed orientation; a secondfilter having a variable orientation to allow for adjustment oftransmission of light; wherein the first and second filters are stackedin series in one or more of the illumination path and optical path.

9. The apparatus of embodiment 5: wherein the housing further comprisesan optical tube; the optical tube having a central aperture; the opticaltube being attached to the housing such that the central aperture isaligned with the optical path when the releasable optical assembly isattached to the portable wireless device; wherein the optical tubecomprises a free end extending away from the housing andbuilt-in-camera; wherein the optical fiber comprises one of a bundle ofoptical fibers; the bundle of optical fibers extending from a locationat or near the illumination source to the optical tube wherein theoptical fibers extend axially along the optical tube toward the free endof the tube to form a coaxial layer at least partially surrounding theoptical tube; and wherein the optical fibers are configured to propagatelight from the illumination source in the direction of the free end ofthe optical tube.

10. The apparatus of embodiment 9, further comprising: a speculumconfigured to interface with a body cavity of a patient; the speculumhaving a proximal end and distal free end; said speculum comprising aconical shape that tapers from the proximal end to the free end; theproximal end of the speculum configured to releasably attach to thehousing; said distal end of the speculum having an aperture configuredto concentrically align with the optical tube when the speculum isattached to the housing.

11. The apparatus of embodiment 5: wherein the housing comprises acylindrical tube having a proximal end comprising the attachmentsurface; the cylindrical tube having an open distal end extending awayfrom said portable wireless device when the imaging apparatus isattached to said portable wireless device; wherein the cylindrical tubehas a predetermined length corresponding to an optical characteristic ofsaid lens and said built in camera; and wherein the open distal end ofthe cylindrical body is configured to be positioned to contact a surfaceof a patient's skin to facilitate imaging of said skin surface.

12. The apparatus of embodiment 11: wherein the optical fiber comprisesone of a bundle of optical fibers; further comprising a diffuserdisposed between the optical fibers and the open distal end of thecylindrical tube; and wherein the diffuser is configured to diffuselight propagated through the bundle of optical fibers.

13. The apparatus of embodiment 5, further comprising: a calibrationmodule attached to said housing; the calibration module being disposedin a field of view of the optical path to provide calibration of thebuilt in camera.

14. The apparatus of embodiment 1, further comprising: a base member;wherein the base member comprises an attachment surface for coupling thebase member to the wireless device; wherein the base member isconfigured to releasably attach to the housing; and wherein the basemember comprises at least one aperture aligned with the illuminationpath and optical path for imaging and illumination through the housing.

15. A system for enhancing and post-processing images obtained from aportable wireless device having a built-in camera, comprising: areleasable optical assembly, comprising: a housing; the housingcomprising an attachment surface for releasably coupling the releasableoptical assembly to the portable wireless device; an opticaltransmission element; wherein the optical transmission element isconfigured to enhance an image taken by the built-in camera prior to theimage being received the portable wireless device; programmingexecutable on said wireless device or other external device for:receiving the enhanced image; post processing the enhanced image.

16. The system of embodiment 15, wherein post processing comprises imagenormalization.

17. The system of embodiment 16, wherein image normalization comprises:calculating one or more of exposure and focus characteristics with aknown feature; and adjusting the image based on reference data from theimage.

18. The system of embodiment 16, wherein said programming is furtherconfigured for calibrating said built in camera as a function of opticalcharacteristics releasable optical assembly.

19. The system of embodiment 17, wherein said programming is furtherconfigured for: inputting data from the normalized image along withother image data; and comparing the data from the normalized image withthe other image data to analyze said image.

20. The system of embodiment 19, wherein the other image data comprisesimages from similar images to the normalized image.

21. The system of embodiment 20, wherein the normalized image comprisesan image of a person's skin; and wherein the other image data comprisesskin images from patients having similar skin characteristics.

22. The system of embodiment 19, wherein said programming is furtherconfigured for: transmitting said image to a secondary device to performpost processing.

23. The system of embodiment 15, wherein the optical transmissionelement comprises a first optical transmission element configured tomagnify the image and a second optical transmission element configuredto enhance illumination of the image.

24. The system of embodiment 23: wherein the first optical transmissionelement comprises a lens; the housing configured to house the lens in analignment within an optical path of the built-in camera; the portablewireless device further comprising an illumination source; wherein thesecond optical transmission element comprises an optical fiber; thehousing configured to house the optical fiber in an alignment within theillumination path of the illumination source; and wherein the opticalfiber is configured to alter the illumination path to enhanceillumination of the image.

25. The system of embodiment 24: the releasable optical assembly furthercomprising one or more optical filters coupled to the housing; and thehousing configured to house the one or more optical filters in analignment within the one or more of the illumination path or opticalpath.

26. The system of embodiment 25, wherein the one or more optical filterscomprises a filter comprising one or more of a: polarizer, neutraldensity filter, or spectral selection filter.

27. The system of embodiment 24: wherein the housing further comprisesan optical tube; the optical tube having a central aperture; the opticaltube being attached to the housing such that the central aperture isaligned with the optical path when the releasable optical assembly isattached to the portable wireless device; wherein the optical tubecomprises a free end extending away from the housing andbuilt-in-camera; wherein the optical fiber comprises one of a bundle ofoptical fibers; the bundle of optical fibers extending from a locationat or near the illumination source to the optical tube wherein theoptical fibers extend axially along the optical tube toward the free endof the tube to form a coaxial layer at least partially surrounding theoptical tube; and wherein the optical fibers are configured to propagatelight from the illumination source in the direction of the free end ofthe optical tube.

28. The system of embodiment 27, further comprising: a speculumconfigured to interface with a body cavity of a patient; the speculumhaving a proximal end and distal free end; said speculum comprising aconical shape that tapers from the proximal end to the free end; theproximal end of the speculum configured to releasably attach to thehousing; said distal end of the speculum having an aperture configuredto concentrically align with the optical tube when the speculum isattached to the housing.

29. The system of embodiment 24: wherein the housing comprises acylindrical tube having a proximal end comprising the attachmentsurface; the cylindrical tube having an open distal end extending awayfrom said portable wireless device when the imaging apparatus isattached to said portable wireless device; wherein the cylindrical tubehas a predetermined length corresponding to an optical characteristic ofsaid lens and said built in camera; and wherein the open distal end ofthe cylindrical body is configured to be positioned to contact a surfaceof a patient's skin to facilitate imaging of said skin surface.

30. A method for enhancing and post-processing images obtained from aportable wireless device having a built-in camera, comprising: receivingan image from a built-in camera of a portable wireless device; whereinthe portable wireless device comprises a releasable optical assemblyreleasably disposed within a field of view of the built-in camera;wherein the releasable optical assembly comprises an opticaltransmission element configured to enhance the image prior to the imagebeing received by the portable wireless device; and post processing theenhanced image.

31. The method of embodiment 30, further comprising: transmitting theenhanced image to a secondary device for post processing.

32. The method of embodiment 30, wherein post processing comprisesnormalizing the image.

33. The method of embodiment 32, wherein normalizing the imagecomprises: calculating one or more of exposure and focus characteristicswith a known feature; and adjusting the image based on reference datafrom the image.

34. The method of embodiment 30, further comprising: calibrating saidbuilt in camera as a function of optical characteristics releasableoptical assembly.

35. The method of embodiment 33, further comprising: inputting data fromthe normalized image along with other image data; and comparing the datafrom the normalized image with the other image data to analyze saidimage.

36. The method of embodiment 35, wherein the other image data comprisesimages from similar images to the normalized image.

37. The method of embodiment 35, wherein the normalized image comprisesan image of a person's skin; and wherein the other image data comprisesskin images from patients having similar skin characteristics.

38. The method of embodiment 30, wherein the optical transmissionelement comprises a first optical transmission element configured tomagnify the image and a second optical transmission element configuredto enhance illumination of the image. Although the description abovecontains many details, these should not be construed as limiting thescope of the invention but as merely providing illustrations of some ofthe presently preferred embodiments of this invention.

Therefore, it will be appreciated that the scope of the presentinvention fully encompasses other embodiments which may become obviousto those skilled in the art, and that the scope of the present inventionis accordingly to be limited by nothing other than the appended claims,in which reference to an element in the singular is not intended to mean“one and only one” unless explicitly so stated, but rather “one ormore.” All structural, chemical, and functional equivalents to theelements of the above-described preferred embodiment that are known tothose of ordinary skill in the art are expressly incorporated herein byreference and are intended to be encompassed by the present claims.Moreover, it is not necessary for a device or method to address each andevery problem sought to be solved by the present invention, for it to beencompassed by the present claims. Furthermore, no element, component,or method step in the present disclosure is intended to be dedicated tothe public regardless of whether the element, component, or method stepis explicitly recited in the claims. No claim element herein is to beconstrued as a “means plus function” element unless the element isexpressly recited using the phrase “means for”. No claim element hereinis to be construed as a “step plus function” element unless the elementis expressly recited using the phrase “step for”.

What is claimed is:
 1. A method for enhancing and post-processing imagesobtained from a portable wireless device having a built-in camera,comprising: receiving an image from the built-in camera of the portablewireless device; wherein the portable wireless device comprises areleasable optical assembly releasably disposed within a field of viewof the built-in camera; wherein the releasable optical assemblycomprises an optical transmission element configured to enhance theimage prior to the image being received by the portable wireless device;and post processing the enhanced image.
 2. A method as recited in claim1, further comprising: transmitting the enhanced image to a secondarydevice for post processing.
 3. A method as recited in claim 1, whereinpost processing comprises normalizing the image.
 4. A method as recitedin claim 3, wherein normalizing the image comprises: calculating one ormore of exposure and focus characteristics with a known feature; andadjusting the image based on reference data from the image.
 5. A methodas recited in claim 1, further comprising: calibrating said built incamera as a function of optical characteristics releasable opticalassembly.
 6. A method as recited in claim 4, further comprising:inputting data from the normalized image along with other image data;and comparing the data from the normalized image with the other imagedata to analyze said image.
 7. A method as recited in claim 6, whereinthe other image data comprises images from similar images to thenormalized image.
 8. A method as recited in claim 6, wherein thenormalized image comprises an image of a person's skin; and wherein theother image data comprises skin images from patients having similar skincharacteristics.
 9. A method as recited in claim 1, wherein the opticaltransmission element comprises a first optical transmission elementconfigured to magnify the image and a second optical transmissionelement configured to enhance illumination of the image.
 10. A methodfor enhancing and post-processing images obtained from a portablewireless device having a built-in camera, comprising: (a) receiving animage from the built-in camera of the portable wireless device; (b)wherein the portable wireless device comprises a releasable opticalassembly releasably disposed within a field of view of the built-incamera; (c) wherein the releasable optical assembly comprises an opticaltransmission element configured to enhance the image prior to the imagebeing received by the portable wireless device; (d) post processing theenhanced image; and (e) transmitting the enhanced image to a secondarydevice for post processing; (f) wherein post processing comprisesnormalizing the image; and (g) wherein normalizing the image comprises:(i) calculating one or more of exposure and focus characteristics with aknown feature; and (ii) adjusting the image based on reference data fromthe image.
 11. A method as recited in claim 10, further comprising:calibrating said built in camera as a function of opticalcharacteristics releasable optical assembly.
 12. A method as recited inclaim 10, further comprising: inputting data from the normalized imagealong with other image data; and comparing the data from the normalizedimage with the other image data to analyze said image.
 13. A method asrecited in claim 12, wherein the other image data comprises images fromsimilar images to the normalized image.
 14. A method as recited in claim12: wherein the normalized image comprises an image of a person's skin;and wherein the other image data comprises skin images from patientshaving similar skin characteristics.
 15. A method as recited in claim10, wherein the optical transmission element comprises a first opticaltransmission element configured to magnify the image and a secondoptical transmission element configured to enhance illumination of theimage.
 16. A method for enhancing and post-processing images obtainedfrom a portable wireless device having a built-in camera, comprising:(a) receiving an image from the built-in camera of the portable wirelessdevice; (b) wherein the portable wireless device comprises a releasableoptical assembly releasably disposed within a field of view of thebuilt-in camera; (c) wherein the releasable optical assembly comprisesan optical transmission element configured to enhance the image prior tothe image being received by the portable wireless device; (d) postprocessing the enhanced image; (e) transmitting the enhanced image to asecondary device for post processing. (f) wherein post processingcomprises normalizing the image by calculating one or more of exposureand focus characteristics with a known feature and adjusting the imagebased on reference data from the image; (g) calibrating said built incamera as a function of optical characteristics releasable opticalassembly; (h) inputting data from the normalized image along with otherimage data; and (i) comparing the data from the normalized image withthe other image data to analyze said image; (j) wherein the other imagedata comprises images from similar images to the normalized image; (k)wherein the normalized image comprises an image of a person's skin; and(l) wherein the other image data comprises skin images from patientshaving similar skin characteristics.
 17. A method as recited in claim16, wherein the optical transmission element comprises a first opticaltransmission element configured to magnify the image and a secondoptical transmission element configured to enhance illumination of theimage.